University of Kansas - Machine and Statistical Learning
Published
May 19, 2026
Modified
May 19, 2025
Introduction
About one out of every three adults in the United States have reported not seeking health care because of the cost (Sparks et. al. 2026). I explored whether we could predict whether individuals would avoid seeking medical care based on health status, socioeconomic, and demographic indicators. I utilized publicly available data to train machine learning models in a classification framework. This research hopes to test the validity of predictive modeling on this topic as well as gain additional insights into this phenomenon.
Data Description and Preprocessing
Data was collected from the Behavioral Risk Factor Surveillance System database maintained by the CDC. This is an annual telephone survey that focuses on health status and healthcare utilization metrics across the United States. Data used in this project was harmonized from the 2017-2024 survey years, resulting in about 3.5 million observations before preprocessing. Table 1 shows main variable groups for prediction (for full data dictionary, see appendix - table a1)
Due to high frequency of missing data in predictors, I implemented a one-hot coding method to impute and track observations with missing values in binary predictors. Then, dropping missing rows removed 500,401 rows, resulting in 2,966,543 rows in the final dataset. All variables were categorical and were therefore factored and labeled accordingly. Exploratory data analysis shows the severe class imbalance on the outcome variable with only 8% of observations being true positive cases (Figure 1).
Show code
ggplot(full_df, aes(x = MEDCOST, fill = MEDCOST)) +geom_bar(aes(y =after_stat(count)), fill ="steelblue") +scale_y_continuous(labels = scales::label_comma()) +labs(y ="Total observations",x ="MEDCOST",title ="Severe class imbalance in outcome" ) +annotate(geom ="rect", fill ="white",alpha =0.7,xmin =0.5, ymin =1.1e6,xmax =1.5, ymax =1.9e6 ) +annotate(geom ="text",x =1, y =1.7e6,label ="Class proportions:" ) +annotate(geom ="text", x =1, y =1.5e6, label ="Positive (Avoided care): 0.08" ) +annotate(geom ="text",x =1, y =1.3e6,label ="Negative (Did not avoid care): 0.92" )
Figure 1: Histogram showing the extreme class imbalance in outcome variable
Methods
I tested four models using a caret workflow: a ridge and lasso penalized regression for feature selection and shrinkage, a decision tree as an interpretive baseline, and random forest to reduce overfitting and improve generalizability over the decision tree.
3-fold cross validation with a train/test split was conducted on all models except for the random forest, due to computational restrictions. As a result of this, the random forest model was evaluated using out-of-bag predictions. Hyperparameter tuning was conducted on all models to identify the best parameters based on ROC AUC performance.
Table B. Penalized regression tuning parameters
Model
Hyperparameter
Low bound
High bound
Increment
Length
Ridge
lambda
0.001
10
10
alpha = 0
threshold
0.005
0.95
0.1
Lasso
lambda
0.001
10
10
alpha = 1
threshold
0.005
0.95
0.1
Table C. Tree-based model tuning parameters
Model
Hyperparameter
Values
Decision Tree
Complexity Parameter (CP)
1e-5, 3e-5, 1e-4
Threshold
0.005 - 0.95
Random Forest
mtry
6
min.node.size
20
split.mode
gini
threshold
0.005 - 0.95
Results
All models showed moderate ranking ability between both cross-validated and hold-out data (ROC AUC = 0.773-0.817). Balanced accuracy was selected as the primary evaluation metric to account for the severe class imbalance in the outcome variable. Sensitivity and specificity were included to evaluate discrimination between cases. (Table 1)
* RF model uses out-of-bag (OOB) probabilities instead of CV evaluation on training. While OOB is analogous to CV, predictions for OOB are likely more optimistic
The decision tree model was the most unstable of all of the models, showing a noticeably lower ROC AUC for the cross-validated set (0.773) than the test set (0.814). Additionally, this model shows a lower sensitivity for and a higher specificity for the CV set than the test set, indicating instability. This phenomenon is visible in Figure 3 and Figure 2.
Show code
mod_comp_long <-read_csv("mod_comp_long.csv")mod_comp_gap <- mod_comp_long %>%pivot_wider(names_from = dataset, values_from = value) %>%mutate(gap = test - cv)mod_comp_gap_plot <-ggplot(mod_comp_gap, aes(x = model, y = gap)) +geom_col() +facet_wrap(~ metric) +labs(title ="CV-Test set gap between key metrics",subtitle ="Smallest bars = most consistent models between training and test set",caption ="*RF model uses OOB probabilities on training; results will be optimistic." )mod_comp_plot <- mod_comp_long %>%ggplot(aes(x = metric, y = value, color = dataset)) +geom_point(size =3, position =position_dodge(width =0.3)) +geom_hline(yintercept =0.5, linetype="dashed") +coord_cartesian(ylim =c(0.50, 0.9)) +facet_wrap(~ model) +labs(y ="Test balanced accuracy",x =NULL,title ="Comparable ranking performance across models",subtitle ="Higher balanced accuracy is better",caption ="*RF model uses OOB probabilities on training; results will be optimistic." )
Figure 2: ML model performance across key metrics
Figure 3: Discrepancy between models on train-test balanced accuracy stability
The Random Forest model is the best performing model based on balanced accuracy at 0.741. However, the Ridge and Lasso models both performed within 0.001 and 0.002 points respectively, so the improvement that the RF model introduces is marginal (Table 1). Additionally, the RF model was not subjected to 3-fold cross validation like the other models so performance was based on out-of-bag predictions, which leads to more optimistic results. Because the Random Forest model did not significantly improve performance over Ridge or Lasso models enough to justify the extra model complexity, Ridge is the best model.
Show code
cm_df <-read_csv("ridge_cm_df.csv")ggplot(cm_df, aes(Reference, Prediction, fill = value)) +geom_tile(color ="white", linewidth =0.5) +geom_text(aes(label = value), color ="white") +scale_fill_gradient(low ="grey60", high ="firebrick") +labs(title ="Ridge GLM confusion matrix at threshold = 0.08")
Figure 4: Heatmap of the ridge confusion matrix shows predicted outcomes
The model displays moderate recall ability (Sens. = 0.72) but poor precision (0.21) due to the low threshold (0.08) classifying far more cases as likely to avoid care than the default baseline threshold of 0.5. This results in moderately few false negative cases but a high proportion of false positives. A specificity of 0.757 shows moderate performance on predictive negative cases correctly. Due in part to the severe class imbalance and large prevalence of negative cases, the negative predictive value (NPV) is 0.968. This shows that the model is more likely to predict false positives than false negatives. Reducing the probability of false negatives aligns with the goal of not missing vulnerable individuals in predictions, which in turn allows for improved screening and timely interventions. (Figure 4)
Show code
vi <-read_csv("ridge_vi.csv")ggplot(vi, aes(x =reorder(Feature, Overall), y = Overall)) +geom_col(fill ="steelblue") +coord_flip() +labs(x =NULL,y ="Relative importance",title ="Top contributors to predictive performance",caption ="Feature graph ranks predictive contribution, not causality" )
Variable importance shows a number of plausible predictors. The presence of a health plan is the most influential variable for the ridge model. Other socioeconomic and demographic variables, such as age over 65 years and annual income over $200,000 also extend above 50% relative importance, signifying the presence of a signal along those variables. The next bracket, between 25% and 50% relative importance, includes whether one has a personal doctor or primary care physician (PERSDOC_imp) and the prior income bracket (X_INCOMG$100k-200k). It also introduces our first health status indicators for poor majority mental and physical health days (X_MENT14D14+ days; X_PHYS13D14+ days), as well as majority good mental and physical health days (X_MENT14D1-13 days; X_PHYS14D1-13 days). The next age group we see (X_AGE_G25-34) is also in the 15% to 50% bracket. While we cannot infer causality nor magnitude of relationship from the variable importance graph, these findings are consistent with common perceptions about how different age and socioeconomic groups engage with healthcare. (Figure 5)
Ethics & Fairness Considerations
This research exists within the public health and policy domains, but intended use of this research remains strictly educational. Results herein do not imply causality between the phenomena examined. Future works should address the methodological and ethical concerns addressed here.
The data sourced for this research is a publicly-available and anonymized dataset hosted by the CDC (see references for attribution). This data utilizes iterative proportional fitting, also known as raking, to weight survey data representatively to existing proportions of demographic identity within a population. Weighted variables were not used in the development of predictive models and may not be sufficiently representative of individual populations. This is a remaining limitation of this research, and it introduces a bias and fairness risk. This was mitigated by research design avoiding drawing inferences about populations at scale. While the data is depersonalized, there exists a marginal reidentification risk due to the data containing information about demographics, identity, and socioeconomic and health status. To reduce this risk, only summary and aggregate data (confusion matrices, statistical results) were reported.
Microsoft Copilot was used for methodological research, code scaffolding and debugging. Lack of methodological experience introduced a risk of over-reliance on LLM output. This was mitigated by direct code validation and testing by the user as well as cross-reference output with R documentation, course materials, and other sources. No copyrighted material nor raw data was input into the AI assistant, and only summary, aggregate, or results data and original code was used in interpretive or debugging tasks.
Discussion
This research shows that it is possible to predict who is vulnerable to avoid healthcare due to cost from publicly available data. The results indicate that there is a moderate signal that was identified, but further precision could not be attained through model tuning alone. Although the random forest model is far more complex, it did not meaningfully outperform the ridge model. Noise in the signal could stem from unobserved covariates or an inherently weak signal between the predictors and outcome variable.
Another limitation of this project was the scale of the data used in model training. The use of a large dataset introduced a number of computational limitations, and as such, this project lacked a baseline GLM to compare machine learning models against, as well as 3-fold cross-validation on the random forest model. This negatively impacts the validity of this research by weakening the methodological rigor.
Future work should address these limitation by reducing the scale of the data, potentially with a more narrow geographic or temporal scope, to enable more rigorous and thorough cross validation techniques. Additionally, future research should narrow the scope of predictive modeling to variables that display a stronger connection to the phenomenon, such as socioeconomic predictors.
Conclusion
This research shows that comparably simple models like penalized regression can compete with more complex models like random forests on predictive performance. The random forest model showed marginal improvement over the ridge model, the computational and interpretive complexity of the model is not justified by such minor gains.
Through this project, I learned about how to use a caret workflow to cross validate, tune, and train models, as well as how to interpret model performance on new data. I also learned a lot of hard lessons about research design - particularly how I sourced data, developed a research framework, and how the size of data can significantly impact the scope of the project.
References
Data attribution
Centers for Disease Control and Prevention (CDC). Behavioral Risk Factor Surveillance System Survey Data. Atlanta, Georgia: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, 2026.
Works cited
Microsoft. (2026). Intro ML Assistant (GPT‑5–based tuned AI agent for introductory machine learning instruction; customized by course instructor) [Large language model]. https://www.microsoft.com/microsoft-365/copilot
Sparks, G., Lopes, L., Montero, A., Hamel, L. (2026, April 30). Americans’ challenges with health care costs. KFF. https://www.kff.org/health-costs/americans-challenges-with-health-care-costs/
Packages
Friedman J, Hastie T, Tibshirani R (2010). “Regularization Paths for Generalized Linear Models via Coordinate Descent.” Journal of Statistical Software, 33(1), 1-22. doi:10.18637/jss.v033.i01 https://doi.org/10.18637/jss.v033.i01.
Kuhn, M. (2008). Building Predictive Models in R Using the caret Package. Journal of Statistical Software, 28(5), 1–26. https://doi.org/10.18637/jss.v028.i05
Marvin N. Wright, Andreas Ziegler (2017). ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R. Journal of Statistical Software, 77(1), 1-17. doi:10.18637/jss.v077.i01
Tay JK, Narasimhan B, Hastie T (2023). “Elastic Net Regularization Paths for All Generalized Linear Models.” Journal of Statistical Software, 106(1), 1-31. doi:10.18637/jss.v106.i01 https://doi.org/10.18637/jss.v106.i01.
Wickham H, Averick M, Bryan J, Chang W, McGowan LD, François R, Grolemund G, Hayes A, Henry L, Hester J, Kuhn M, Pedersen TL, Miller E, Bache SM, Müller K, Ooms J, Robinson D, Seidel DP, Spinu V, Takahashi K, Vaughan D, Wilke C, Woo K, Yutani H (2019). “Welcome to the tidyverse.” Journal of Open Source Software, 4(43), 1686. doi:10.21105/joss.01686 https://doi.org/10.21105/joss.01686.
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77 http://www.biomedcentral.com/1471-2105/12/77/
